**********************************************
******           Instructions           ******
**********************************************

To replicate the results in, "What can we learn from predictive modeling?" download and unpack replication_archive.tar.gz. The file that runs all of the analysis is in the top level of the replication archive, and is named 'make_analysis.R'. 

Note that there is substantial overlap in the function names in the igraph and sna packages. As such, if you are working with one or the other, running our replication scripts in the same R session may fail due to function overwrites. If you detach all packages in the R session, set the working directory via setwd() to the unpacked replication archive, and then use source() to run make_analysis.R, you will avoid problems with function overwrites. 

The following list describes all of the files in the archive.

The following packages will be needed to run the replication code. The package versions used in the final analysis are listed in quotes.

minet ‘3.28.0’
lda ‘1.3.1’
latentnet '2.4.1'
caTools '1.17.1'
penalized ‘0.9.46’
nnet ‘7.3.12’
PerformanceAnalytics ‘1.4.3541’
gplots ‘3.0.1’
ROCR ‘1.0.7’
countrycode ‘0.18’
zoo ‘1.7.13’
xtable ‘1.8.2’
sna ‘2.3.2’
igraph ‘1.0.1’
RSNNS ‘0.4.7’
lqa ‘1.0.3’
glmnet ‘2.0.5’

**********************************************
******           File List              ******
**********************************************

./Code

AnalyzePredictiveResults.R: Produces all of the AUC analyses and plots comparing the various models and learning methods.

AUC_PR.R: Provides a function for calculating the area under the precision-recall curve.

CovBoost.R: Provides a function for running the covariate-only models with boosting.

CovLogit.R: Provides a function for running the covariate-only models with glmnet regularization.

CovNNET.R: Provides a function for running the covariate-only models with neural network training.

ExampleROC.R: Provides code for the example AUC plots used to illustrate the comparative properties of ROC and PR.

GatherLatent.R: Runs a procedure to organize results from the latent variable models used for covariates (i.e., the LSM and MMSBM).

LogitBoost2.R: Provides a function for running full models with glmnet regularization. 

LogitCoefficientAnalysis.R: runs the analysis comparing glmnet coefficients to logistic regression coefficients.

LSM.R: Provides a function to organize and run the latent space model.

MMSBM.R: Provides a function to organize and run the mixed membership stochastic blockmodel.

NetBoost.R: Provides a function for running the network-only models with boosting.

NetLogit.R: Provides a function for running the network-only models with glmnet regularization.

NetNNET.R: Provides a function for running the network-only models with neural network training.

trainBoost...R: Files following this naming convention train the respective models based on boosting.

trainLogitModels....R: Files following this naming convention train the respective models based on glmnet.

trainNNETModels....R: Files following this naming convention train the respective models based on neural networks.

trainSLogitModels...R: Files following this naming convention train the respective models based on logistic regression. 

./LatentModels/LatentModels##_##.R: Files in the LatentModels directory are used to train the latent variable models for the years denoted by the numerical ranges in the file names. Warning, these scripts take a very long time and a lot of RAM to run. They were cut into separate files for implicit parallelization on a cluster.

./Data

system2008.1.csv: Correlates of War state system membership, 2008 version. 

PreparedData.RData: RData file containing all of the raw data required for the analysis. 

**********************************************
******           Results                ******
**********************************************

Once the code has finished running, the results, reflected as PDF files, will be in the top level of the replication archive. The following list gives the figure for which each file represents the results. Note, those left blank correspond to results files that were included in earlier versions of the paper, but were left out of the final manuscript.

AASim_PrExp.pdf	.......Figure 7	algBar_R.pdf		algBar.pdfallBar_R.pdf..........Figure 3allBar.pdf............Figure 3Allied_PrExp.pdf......Figure 5AllyW_PrExp.pdf.......Figure 5boostOT1_R.pdfboostOT1.pdfboostOT5_R.pdfboostOT5.pdfboostOT10_R.pdfboostOT10.pdfCincRat_PrExp.pdf.....Figure 5ComCmty_PrExp.pdf.....Figure 6CommComb_PrExp.pdf....Figure 7Contig_PrExp.pdf......Figure 6covBar_R.pdfcovBar.pdfFlow_PrExp.pdf........Figure 6InvGeoDist_PrExp.pdf..Figure 6JacSim_PrExp.pdf......Figure 7JntDem_PrExp.pdf......Figure 5lassoOT1_R.pdf........Figure 4lassoOT1.pdf..........Figure 4lassoOT5_R.pdf........Figure 4lassoOT5.pdf..........Figure 4lassoOT10_R.pdf.......Figure 4lassoOT10.pdf.........Figure 4Latent2_PrExp.pdf.....Figure 7Memory_PrExp.pdf......Figure 6MMSBM_PrExp.pdf.......Figure 7MPDyad_PrExp.pdf......Figure 5nnetOT1_R.pdfnnetOT1.pdfnnetOT5_R.pdfnnetOT5.pdfnnetOT10_R.pdfnnetOT10.pdfPRCRug.pdf............Figure 2ROCRug.pdf............Figure 2SharedIGO_PrExp.pdf...Figure 5timeBar_R.pdftimeBar.pdfTradeDep_PrExp.pdf....Figure 6


